Contents

  • 1  Poject
    • 1.1  Project Description
    • 1.2  Libraries
    • 1.3  Custom functions
  • 2  Data loading and general information
    • 2.1  Conclusions
  • 3  Data preprocessing and adding calculations
    • 3.1  Removing complete duplicates
    • 3.2  Replacing data types
    • 3.3  Adding calculations
    • 3.4  Conclusions
  • 4  Data analysis
    • 4.1  Dynamics of events by day
    • 4.2  Clarification of the analyzed period
    • 4.3  Exploring the event funnel
    • 4.4  Analysis of A/B test results
  • 5  General conclusions

Poject¶

Project Description¶

A startup is being considered - an online store selling food. According to the available event log of the mobile application, it is necessary to analyze the actions of users when purchasing goods.

Main tasks:

  • explore the sales funnel,
  • find out how users reach the purchase,
  • how many users reach the purchase, and how many are "stuck" on the previous steps (on which ones),
  • to investigate the results of the A/A/B experiment (control groups 246 and 247 - groups A/A-test, 248 - test group B, in which the font of the interface was changed).

Libraries¶

In [1]:
import pandas as pd
import plotly.express as px
import plotly.graph_objects as go
import scipy.stats as st
import numpy as np

Custom functions¶

In [2]:
def mf_diff(ind):
    """ Сounting the number of users before and after data clipping in the context of a given attribute """
    id_est = (
        df_data.pivot_table(index=ind, aggfunc={'id':'nunique'})
        .join(df_data2.pivot_table(index=ind, aggfunc={'id':'nunique'}), lsuffix='_before',rsuffix='_after')
    )
    id_est = id_est.append(id_est.apply('sum', axis=0).reset_index().set_index('index').rename({0:'Total'}, axis=1).T)
    id_est['diff']=id_est.id_before-id_est.id_after
    id_est['lost_share_proc'] = id_est.apply(lambda x: round(x['diff']*100/x['id_before'], 1) , axis=1)
    id_est.index.name=ind
    return id_est
In [3]:
def mf_bar(df, x_, y_, title, x_text, y_text):
    """ Building a histogram """
    fig=px.bar(df, x=x_, y=y_, text_auto=True)
    fig.update_layout(height=500, width=800, title_text=title)
    fig.update_xaxes(title_text=x_text)
    fig.update_yaxes(title_text=y_text)
    fig.show()
In [4]:
def mf_z_test(goal_list, all_list, alpha=0.05):
    """ Calculation of p-value for binomial distribution (z-test)"""
    
    goal = np.array(goal_list) # conversion numerator
    alll = np.array(all_list) # conversion denominator

    p1 = goal[0]/alll[0] 
    p2 = goal[1]/alll[1] 
    p_combined = (goal[0]+ goal[1])/(alll[0] + alll[1]) 

    difference = p1 - p2  

    z_value = difference / ((p_combined * (1 - p_combined) * (1 / alll[0] + 1 / alll[1]))**0.5)

    distr = st.norm(0, 1) 

    p_value = (1 - distr.cdf(abs(z_value))) * 2

    if p_value < alpha:
        result = 'H1'
    else:
        result = 'H0'

    return [p_value, alpha, result]

Data loading and general information¶

In [5]:
df_data = pd.read_csv('data.csv', sep="\t")

df_data.columns = [x.lower() for x in df_data.columns]
df_data.info()
print(f"\nThe amount of occupied dataset memory: {(df_data.memory_usage('deep')/(1024**2)).sum():.1f} Mb\n")

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 244126 entries, 0 to 244125
Data columns (total 4 columns):
 #   Column          Non-Null Count   Dtype 
---  ------          --------------   ----- 
 0   eventname       244126 non-null  object
 1   deviceidhash    244126 non-null  int64 
 2   eventtimestamp  244126 non-null  int64 
 3   expid           244126 non-null  int64 
dtypes: int64(3), object(1)
memory usage: 7.5+ MB

The amount of occupied dataset memory: 7.5 Mb
In [6]:
# renaming columns
df_data.rename({'eventname':'event', 'deviceidhash':'id', 'eventtimestamp':'ts', 'expid':'gr'}, axis=1, inplace=True)
df_data.head()
Out[6]:
event id ts gr
0 MainScreenAppear 4575588528974610257 1564029816 246
1 MainScreenAppear 7416695313311560658 1564053102 246
2 PaymentScreenSuccessful 3518123091307005509 1564054127 248
3 CartScreenAppear 3518123091307005509 1564054127 248
4 PaymentScreenSuccessful 6217807653094995999 1564055322 248
In [7]:
# analysis of group column values
df_data.gr.value_counts()
Out[7]:
248    85747
246    80304
247    78075
Name: gr, dtype: int64
In [8]:
# checking for complete duplicate strings
print(f"Number of lines - full duplicates: {df_data.duplicated().sum()}")

Number of lines - full duplicates: 413

Conclusions¶

  • column names have been replaced with more convenient ones in lower case,
  • no gaps found in the data,
  • 413 complete duplicates of strings have been detected, they need to be deleted,
  • the column with the date label "ts" will be converted to the date-time type,
  • the column with the group number "gr" has only three non-negative integer values, will be converted to the unsigned type.

Data preprocessing and adding calculations¶

Removing complete duplicates¶

In [9]:
df_data.drop_duplicates(inplace=True)
print(f"Number of lines - full duplicates: {df_data.duplicated().sum()}")

Number of lines - full duplicates: 0

Replacing data types¶

In [10]:
# the date of the event is converted to the date-time type
df_data.ts = pd.to_datetime(df_data.ts, unit='s')

# the index of the group is converted to an integer
df_data.gr = pd.to_numeric(df_data.gr, downcast='unsigned')
In [11]:
# the final structure of the dataframe after conversion
print(f"\nThe amount of occupied dataset memory after conversion: {(df_data.memory_usage('deep')/(1024**2)).sum():.1f} Mb\n")
df_data.info()

The amount of occupied dataset memory after conversion: 7.7 Mb

<class 'pandas.core.frame.DataFrame'>
Int64Index: 243713 entries, 0 to 244125
Data columns (total 4 columns):
 #   Column  Non-Null Count   Dtype         
---  ------  --------------   -----         
 0   event   243713 non-null  object        
 1   id      243713 non-null  int64         
 2   ts      243713 non-null  datetime64[ns]
 3   gr      243713 non-null  uint8         
dtypes: datetime64[ns](1), int64(1), object(1), uint8(1)
memory usage: 7.7+ MB

Adding calculations¶

In [12]:
# a separate date column is created
df_data['dt'] = df_data['ts'].dt.date

Conclusions¶

  • full duplicates of strings removed from dataset,
  • the date column is converted to the date-time type,
  • a separate date column "dt" has been created,
  • the group column has been converted to uint8.

Data analysis¶

In [13]:
# total events in the data
df_data.event.value_counts().reset_index().rename({'index':'Event','event':'Number of events'}, axis=1)
Out[13]:
Event Number of events
0 MainScreenAppear 119101
1 OffersScreenAppear 46808
2 CartScreenAppear 42668
3 PaymentScreenSuccessful 34118
4 Tutorial 1018
In [14]:
# number of unique users
print(f"Number of unique users: {len(df_data.id.unique())}")

Number of unique users: 7551
In [15]:
# average number of events per user
print("Median quantity " +
      f"events per user: {df_data.pivot_table(index='id', aggfunc={'event':'count'}).median()[0]:.0f}")

Median quantity events per user: 20

Intermediate conclusions

  • there are 5 events in the data, judging by the name and number of recorded events, MainScreenAppear is the beginning of the funnel, i.e. the first event,
  • on average (median estimate) there are 20 events per user, of which 5 unique events can be repeated,
  • a total of 7551 unique users are represented in the log.

Dynamics of events by day¶

In [16]:
# date range in the data
print(f"The data is presented from {df_data.dt.min()} to {df_data.dt.max()}")

The data is presented from 2019-07-25 to 2019-08-07
In [17]:
# data preparation
temp = (
    df_data
    .pivot_table(index=df_data.ts.dt.round('1h'), aggfunc={'event':'count'})
    .reset_index()
)

# graphics preparation
fig = px.line(temp, x="ts", y="event", title='Dynamics of recorded events')
fig.update_xaxes(title_text="Date")
fig.update_yaxes(title_text="Events number")
fig.show()

Intermediate conclusions

  • data began to be collected from 25.07.2019 to 07.08.2019,
  • the graph of the dynamics of recorded events showed that the main stream of collected statistics began to arrive from 01.08.2019 to 07.08.2019, for further analysis we will limit ourselves to the data of this period.

Clarification of the analyzed period¶

In [18]:
# days with incomplete data are cut off
df_data2 = df_data.query("ts >= '2019-08-01'")
print(f"Number of unique users after data clipping: {df_data2.id.nunique()}")

Number of unique users after data clipping: 7534
In [19]:
# estimation of data loss by users in the context of events
mf_diff('event')
Out[19]:
id_before id_after diff lost_share_proc
event
CartScreenAppear 3749 3734 15 0.4
MainScreenAppear 7439 7419 20 0.3
OffersScreenAppear 4613 4593 20 0.4
PaymentScreenSuccessful 3547 3539 8 0.2
Tutorial 847 840 7 0.8
Total 20195 20125 70 0.3
In [20]:
# estimation of data loss by users in the context of groups
mf_diff('gr')
Out[20]:
id_before id_after diff lost_share_proc
gr
246 2489 2484 5 0.2
247 2520 2513 7 0.3
248 2542 2537 5 0.2
Total 7551 7534 17 0.2

Intermediate conclusions

  • initially, there were logs for 7551 unique users in the dataset, after the data was cut off, the indicator decreased to 7534, i.e. 17 fewer users,
  • in general, the decrease in the number of unique users in the context of events lies in the range from 0.2% to 0.8% of their original number, and by groups - from 0.2% to 0.3%, i.e. data losses are negligible.

Exploring the event funnel¶

In [21]:
# data preparation
grd1 = (
    df_data2
    .pivot_table(index='event', aggfunc={'event':'count'})
    .rename({'event':'count'}, axis=1)
    .reset_index()
    .sort_values(by='count', ascending=False)
)
In [22]:
# graphics preparation
fig=px.pie(grd1, names='event', values='count')
fig.update_layout(height=500, width=500, title_text="Shares of events in the total volume of all events")
fig.show()
In [23]:
mf_bar(grd1, 'event', 'count', 'Histogram of the number of events','Events', 'Events number ')

Intermediate conclusions

  • interpretation of events:
    • 'MainScreenAppear' - showing the main screen,
    • 'OffersScreenAppear' - showing the screen of offers (products),
    • 'CartScreenAppear' - showing the cart screen,
    • 'PaymentScreenSuccessful - showing the successful payment screen',
    • 'Tutorial' - showing the manual screen , - we exclude the 'Tutorial' event from the funnel analysis, since it does not directly affect the course of product sales, so out of 20125 events, only 1005 relate to 'Tutorial', which is approximately 0.5% of the events under consideration.
In [24]:
# data preparation without Tutorial stage
df_data3 = df_data.query("ts >= '2019-08-01' and event != 'Tutorial'")
In [25]:
# counting the number of unique users at each step of the funnel
grd2 = (
    df_data3.pivot_table(index='event', aggfunc={'id':'nunique'})
    .reset_index()
    .rename({'id':'count'}, axis=1)
    .sort_values(by='count', ascending=False)
    .reset_index(drop=True)
)

# counting conversions to the previous event
grd2['shift'] = (
    grd2['count']
    .shift(periods=1, axis=0)
)

grd2['conversion'] = grd2.apply(lambda x: round(x['count']*100/x['shift'],1), axis=1)
grd2['delta'] = grd2['shift'] - grd2['count']
grd2
Out[25]:
event count shift conversion delta
0 MainScreenAppear 7419 NaN NaN NaN
1 OffersScreenAppear 4593 7419.0 61.9 2826.0
2 CartScreenAppear 3734 4593.0 81.3 859.0
3 PaymentScreenSuccessful 3539 3734.0 94.8 195.0
In [26]:
# # building a food sales funnel
fig = go.Figure(
    go.Funnel(y=list(grd2.reset_index()['event']), x=list(grd2.reset_index()['count'])
        ))
fig.update_layout(height=500, width=950, title = "Food sales funnel on the website")    
fig.update_yaxes(title_text="Funnel Steps") 
    
fig.show()

Intermediate conclusions

  • the largest number of unique users is lost when switching from the MainScreenAppear event (main screen) to OffersScreenAppear (product offer screen), the conversion of the transition from the main screen to the product offer screen was 62%,
  • on the following transitions, the conversion rate to the previous step increases (OffersScreenAppear -> CartScreenAppear: 81% and CartScreenAppear -> PaymentScreenSuccessful 95%), i.e. the closer buyers are to paying for the formed basket, the greater the proportion of successful transitions to the next step,
  • the final conversion of the transition from the main screen to the payment screen (MainScreenAppear -> PaymentScreenSuccessful) was 48%, i.e. half of the users who got to the main screen will make a purchase with a probability of almost 50%.

Analysis of A/B test results¶

In [27]:
# checking for the flow of users from group to group
print("The number of users who switched to other groups during the test: "+
      f"{df_data3.pivot_table(index='id', aggfunc={'gr':'nunique'}).query('gr>1').count()[0]}")

The number of users who switched to other groups during the test: 0
In [28]:
# number of users participating in the A/B test
grd=df_data3.pivot_table(index='gr', aggfunc={'id':'nunique'}).reset_index()
mf_bar(grd, 'gr', 'id', 'Number of unique groups by A/B-test groups', 'Test group number', 'Number of users')
In [29]:
print("Checking the equality of the number of users by test groups:")
print(f"246/247: {grd.set_index('gr').loc[246, 'id']/grd.set_index('gr').loc[247, 'id']:.3f}")
print(f"247/248: {grd.set_index('gr').loc[247, 'id']/grd.set_index('gr').loc[248, 'id']:.3f}")
print(f"246/248: {grd.set_index('gr').loc[246, 'id']/grd.set_index('gr').loc[248, 'id']:.3f}")

Checking the equality of the number of users by test groups:
246/247: 0.988
247/248: 0.991
246/248: 0.979

Intermediate conclusions

  • no users who switched to other groups during the test were found, i.e. the splitting of users into groups was performed correctly,
  • the number of unique users in the group 246 (first test) - 2483 people, 247 (second test) - 2515, 248 (control) - 2535,
  • the value of the indicator in all groups should be the same, we have small differences: 246/247 = 0.988, 247/248 = 0.991, 246/248 = 0.979,
  • the differences in the number of users in groups should not exceed 1%, here it is slightly more, which indicates that the users are not divided into groups of sufficient quality, or these users were lost due to data clipping by date,
  • in general, we assume that the number of users in groups is the same, which will allow us to accept the results of A/B test.
In [30]:
# sales funnel by three groups
grd= (
    df_data3
    .pivot_table(index='event', columns='gr', aggfunc={'id':'nunique'})
    .droplevel(level=0, axis=1)
    .sort_values(by=246, ascending=False)
)
grd
Out[30]:
gr 246 247 248
event
MainScreenAppear 2450 2476 2493
OffersScreenAppear 1542 1520 1531
CartScreenAppear 1266 1238 1230
PaymentScreenSuccessful 1200 1158 1181
In [31]:
# building a segmented funnel
fig = go.Figure()
for i in grd.columns:  
    fig.add_trace(go.Funnel(
        name = i,
        y = list(grd.index),
        x = list(grd.loc[:,i])
        )) 
fig.update_layout(title = "Funnel of food sales on the website by A/B test groups")    
fig.update_yaxes(title_text="Funnel Steps") 
fig.show()

A/B test results analysis plan

  • we will perform statistical hypothesis testing based on A/A and A/B test results in combinations, groups:
  • group 246 with group 247 (A1/A2 test)
  • 246 with 248 (A1/B test),
    • 247 with 248 (A2/B test),
      • 246 and 247 with 248 (A 12/B test),
  • z-test is used to test statistical hypotheses when comparing fractions,
    • for the null hypothesis, let's take the statement: for the selected step of the funnel, the shares of unique users from all unique users (conversion) by groups coincide,
  • alternative hypothesis: for the selected step of the funnel, the shares of users by groups differ,
  • since the same data will be repeatedly used to test the stat hypotheses, i.e. multiple comparisons are performed, then it is necessary to calculate the correction of the stat significance level.
In [32]:
# data preparation - the number of unique users for each group by funnel steps
df_t= df_data3.pivot_table(index='gr', columns='event', aggfunc={'id':'nunique'}).droplevel(level=0, axis=1)

# adding combined data by 246 and 247 groups and sorting by funnel steps
df_t = (
    df_t.append(
        df_data3.query("gr in (246,247)")
        .pivot_table(index='event', aggfunc={'id':'nunique'})
        .rename({'id':'246_247'}, axis=1)
        .T
    )
    .T
    .sort_values(by=246, ascending=False)
    .T
)

# adding totals by groups
df_t['total']=(
    df_data3
    .pivot_table(index='gr', aggfunc={'id':'nunique'})
    .append(pd.DataFrame(df_data3.query("gr in (246,247)")['id'].nunique(), columns=['id'], index=['246_247']))
    .rename({'id':'total'}, axis=1)
)
    
print("The number of unique users by groups and funnel steps:")
df_t

The number of unique users by groups and funnel steps:
Out[32]:
event MainScreenAppear OffersScreenAppear CartScreenAppear PaymentScreenSuccessful total
246 2450 1542 1266 1200 2483
247 2476 1520 1238 1158 2512
248 2493 1531 1230 1181 2535
246_247 4926 3062 2504 2358 4995
In [33]:
# checking user shares by funnel steps by groups
print("Checking user shares by funnel steps by groups as a percentage:")
df_t.apply(lambda x: x*100/x['total'], axis=1)

Checking user shares by funnel steps by groups as a percentage:
Out[33]:
event MainScreenAppear OffersScreenAppear CartScreenAppear PaymentScreenSuccessful total
246 98.670963 62.102296 50.986710 48.328635 100.0
247 98.566879 60.509554 49.283439 46.098726 100.0
248 98.343195 60.394477 48.520710 46.587771 100.0
246_247 98.618619 61.301301 50.130130 47.207207 100.0
In [34]:
# threshold level of stat significance
alpha = 0.05

# Shidak's correction for 4 funnel steps and 4 comparisons
alpha_sh = 1-(1-alpha)**(1/(4*4))
print(f"Threshold level of significance after the Shidak amendment: {alpha_sh:.2%}")

Threshold level of significance after the Shidak amendment: 0.32%
In [35]:
# calculation of all stat tests
voc1={'246 to 247 (А1/А2)':[246, 247], '246 to 248 (А1/В)':[246, 248]
      , '247 to 248 (А2/В)':[247, 248], '246 & 247 to 248 (А12/В)':['246_247', 248]}
for j in voc1:
    groups = voc1[j]
    print(f"\n\n----  Groups comparison {j} by funnel steps:")
    for i in df_t.columns:
        if i !='total':
            res = mf_z_test([df_t.loc[groups[0], i], df_t.loc[groups[1], i]]
                      , [df_t.loc[groups[0], 'total'], df_t.loc[groups[1], 'total']], alpha_sh)
            if res[2] == 'H0':
                text = f"the null hypothesis is not rejected (the shares are the same) at the significance level {res[0]:.1%}"
            else:
                text = f"an alternative hypothesis was adopted (the shares are different) at the level of significance {res[0]:.1%}"
            print(f"{i}:  \n    "+text)


----  Groups comparison 246 to 247 (А1/А2) by funnel steps:
MainScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 75.3%
OffersScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 24.8%
CartScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 22.9%
PaymentScreenSuccessful:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 11.4%


----  Groups comparison 246 to 248 (А1/В) by funnel steps:
MainScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 33.9%
OffersScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 21.4%
CartScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 8.1%
PaymentScreenSuccessful:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 21.7%


----  Groups comparison 247 to 248 (А2/В) by funnel steps:
MainScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 51.9%
OffersScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 93.3%
CartScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 58.8%
PaymentScreenSuccessful:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 72.8%


----  Groups comparison 246 & 247 to 248 (А12/В) by funnel steps:
MainScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 34.9%
OffersScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 44.6%
CartScreenAppear:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 18.7%
PaymentScreenSuccessful:  
    the null hypothesis is not rejected (the shares are the same) at the significance level 61.1%

Intermediate conclusions

  • at the first step of the MainScreenAppear funnel, the share of users in all groups was slightly more than 98%, which indicates that some users bypassing it get to the next steps (for example, if a person immediately found the right product after searching and went to the shopping cart or order payment),
  • the value of the critical level of stat readability (alpha) was selected 5%, with the correction of Shidak, an indicator was adopted for 16 comparisons, the value of which was 0.32%,
  • the difference in user shares (conversions) by funnel steps for groups 246 and 247 (A/A test) is insignificant, which is a necessary condition for the correctness of data processing for A/B test:
  • for the MainScreenAppear step for group 246, the share was 98.7%, for 247 - 98.6%,
  • OffersScreenAppear - 62% and 60.5%,
  • CartScreenAppear - 51% and 49%,
    • PaymentScreenSuccessful - 48.3% and 46.1%,
  • no statistically significant difference in funnel steps in groups 246 and 247 was found in the A/A test, which is a prerequisite for technical verification of A/B test data collection, so:
    • the level of stat significance of the MainScreenAppear step was 75.3%,
  • OffersScreenAppear - 24.8%,
  • CartScreenAppear - 22.9%,
  • PaymentScreenSuccessful - 11.4%,
  • when comparing groups 246 and 248 by funnel steps, no statistically significant difference was found at the corresponding significance levels (MainScreenAppear - 33.9%, OffersScreenAppear - 21.4%, CartScreenAppear - 8.1%, PaymentScreenSuccessful - 21.7%),
  • when comparing groups 247 and 248 by funnel steps, no statistically significant difference was found at the corresponding significance levels (MainScreenAppear - 51.9%, OffersScreenAppear - 93.3%, CartScreenAppear - 58.8%, PaymentScreenSuccessful - 72.8%),
  • when comparing groups 246 + 247 and 248 by funnel steps, no statistically significant difference was found at the corresponding significance levels (MainScreenAppear - 34.9%, OffersScreenAppear - 44.6%, CartScreenAppear - 18.7%, PaymentScreenSuccessful - 61.1%),
  • from the results of statistical tests for all steps of the funnel, not a single null hypothesis was rejected, the minimum significance level was 8.1%, which is significantly higher than the threshold value of 0.32%,
  • during the A/B test for each step of the product funnel, no statistically significant differences were found for any group, i.e. the change in the font size in the test group 248 did not affect the conversion rate in any way compared to the control groups.

General conclusions¶

Preprocessing and adding calculations

  • column names have been replaced with more convenient ones in lower case,
  • no gaps found in the data,
  • 413 full duplicates of lines were detected, were deleted,
  • the date column is converted to the date-time type,
  • a separate date column "dt" has been created,
  • the group column has been converted to uint8.

Data analysis

  • interpretation of events:
    • 'MainScreenAppear' - showing the main screen,
    • 'OffersScreenAppear' - showing the screen of offers (products),
    • 'CartScreenAppear' - showing the cart screen,
    • 'PaymentScreenSuccessful - showing the successful payment screen',
    • 'Tutorial' - showing the manual screen,
  • 5 events are presented in the data: based on the name of the event and the analysis of the number of events, MainScreenAppear is the first event,
  • the number of Tutorial events turned out to be negligible and it was excluded from the analysis,
  • the data in the logs were collected from 25.07.2019 to 07.08.2019, however, the main stream of collected statistics began to arrive from 01.08.2019 to 07.08.2019, which was taken as a working one,
  • initially, there were logs for 7551 unique users in the dataset, after the data was cut off, the indicator decreased to 7534, i.e. 17 fewer users,
  • in general, the decrease in the number of unique users in the context of events lies in the range from 0.2% to 0.8% of their original number, and by groups - from 0.2% to 0.3%, i.e. data losses are negligible,
  • the largest number of unique users is lost when switching from the MainScreenAppear event (main screen) to OffersScreenAppear (product offer screen), the conversion of the transition from the main screen to the product offer screen was 62%,
  • on the following transitions, the conversion rate to the previous step increases (OffersScreenAppear -> CartScreenAppear: 81% and CartScreenAppear -> PaymentScreenSuccessful 95%), i.e. the closer buyers are to paying for the formed basket, the greater the proportion of successful transitions to the next step,
  • the final conversion of the transition from the main screen to the payment screen (MainScreenAppear -> PaymentScreenSuccessful) was 48%, i.e. half of the users who got to the main screen will make a purchase with a probability of almost 50%.

Analysis of A/B test results

  • no users who switched to other groups during the test were found, i.e. the splitting of users into groups was performed correctly,
  • the number of unique users in the group 246 (first test) - 2483 people, 247 (second test) - 2515, 248 (control) - 2535,
  • the value of the indicator in all groups should be the same, we have small differences: 246/247 = 0.988, 247/248 = 0.991, 246/248 = 0.979,
  • the differences in the number of users in groups should not exceed 1%, here it is slightly more, which indicates that the users are not divided into groups of sufficient quality, or these users were lost due to data clipping by date,
  • in general, we assume that the number of users in groups is the same, which will allow us to accept the results of A/B test,
  • z-test was used to test statistical hypotheses when comparing fractions,
  • for the null hypothesis, the statement is taken: for the selected funnel step, the shares of unique users from all unique users (conversion) by groups coincide,
  • alternative hypothesis: for the selected funnel step, the shares of users by groups differ,
  • at the first step of the MainScreenAppear funnel, the share of users in all groups was slightly more than 98%, which indicates that some users bypassing it get to the next steps (for example, if a person immediately found the right product after searching and went to the shopping cart or order payment),
  • the value of the critical level of stat readability (alpha) was selected 5%, with the correction of Shidak, an indicator was adopted for 16 comparisons, the value of which was 0.32%,
  • the difference in user shares (conversions) by funnel steps for groups 246 and 247 (A/A test) is insignificant, which is a necessary condition for the correctness of data processing for A/B test:
  • for the MainScreenAppear step for group 246, the share was 98.7%, for 247 - 98.6%,
  • OffersScreenAppear - 62% and 60.5%,
  • CartScreenAppear - 51% and 49%,
    • PaymentScreenSuccessful - 48.3% and 46.1%,
  • in the A/A test, no statistically significant difference in funnel steps was found in groups 246 and 247, which is a prerequisite for technical verification of A/B test data collection, so:
    • the stat level of the significance of the MainScreenAppear step was 75.3%,
  • OffersScreenAppear - 24.8%,
  • CartScreenAppear - 22.9%,
  • PaymentScreenSuccessful - 11.4%,
  • when comparing groups 246 and 248 by funnel steps, no statistically significant difference was found at the corresponding significance levels (MainScreenAppear - 33.9%, OffersScreenAppear - 21.4%, CartScreenAppear - 8.1%, PaymentScreenSuccessful - 21.7%),
  • when comparing groups 247 and 248 by funnel steps, no statistically significant difference was found at the corresponding significance levels (MainScreenAppear - 51.9%, OffersScreenAppear - 93.3%, CartScreenAppear - 58.8%, PaymentScreenSuccessful - 72.8%),
  • when comparing groups 246 + 247 and 248 by funnel steps, no statistically significant difference was found at the corresponding significance levels (MainScreenAppear - 34.9%, OffersScreenAppear - 44.6%, CartScreenAppear - 18.7%, PaymentScreenSuccessful - 61.1%),
  • no null hypothesis was rejected from the results of statistical tests for all funnel steps, the minimum value of the significance level was 8.1%, which is significantly higher than the threshold value of 0.32%,
  • during the A/B test for each step of the product funnel, no statistically significant differences were found for any group, i.e. the font change in the test group 248 did not affect the conversion in any way compared to the control groups.